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I .  INTRODUCTION 


The  Civil  Service  Reform  Act  (CSRA)  of  1978  requires  that  each  Govern¬ 
ment  agency  develop  a  performance-based  employee  appraisal  system.  The 
system  must  permit  accurate  assessment  on  the  basis  of  objective  criteria. 
Within  the  parameters  of  the  law,  the  Air  Force  Human  Resources  Laboratory 
(AFHRL)  has  developed  a  new  approach  to  evaluating  the  job  performance  of 
more  than  200,000  Air  Force  civilian  employees.  The  system  is  called  the 
Job  Pe - formance  Appraisal  System  (JPAS). 

This  system  involves  breaking  each  job  down  into  its  major  components, 
much  the  same  as  is  done  in  preparing  a  position  description.  Task  Analysis 
is  used  to  make  a  list  of  key  work  behaviors  or  practices  essential  to  job 
performance.  Standards  of  performance  are  set  for  each  item  on  the  list, 
and  employees  are  then  rated  against  these  behaviors.  In  order  to  enhance 
employee  participation  and  commitment  to  performance  improvement,  the 
employee  and  supervisor  are  urged  to  jointly  participate  in  the  process. 

The  JPAS  has  three  main  components:  (a)  a  work  plan  which  sets  forth 
the  Job  Performance  Elements  (job  behaviors)  and  associated  performance 
standards;  (b)  a  system  for  reviewing  the  employee's  progress  in  meeting  the 
standards;  and  (c)  a  system  for  rating  the  employee's  performance  relative 
to  the  standards.  While  any  rating  system  is  evaluative  in  nature,  the 
strength  of  this  approach  is  that  the  criteria  against  which  the  employee  is 
assessed  are  very  objective.  That  is,  there  should  be  little  or  no  argument 
about  whether  a  performance  standard  was  met. 

The  JPAS  will  be  used  for  all  civilian  employees  not  identified  as  mem¬ 
bers  of  the  Federal  Senior  Executive  Service  or  General  Manager  employees. 
Therefore,  it  will  be  used  for  all  Federal  Wage  System  (FWS)  and  General 
Schedule  (GS)  employees.  This  means  that  the  same  system  will  be  applied 
across  a  wide  range  of  jobs  at  all  locations  where  civilians  are  employed. 
Figure  1  contains  a  flow  chart  detailing  the  procedures  associated  with 
JPAS . 


The  major  objective  of  this  work  was  to  determine  experimentally  the 
most  efficient,  effective,  and  transportable  approach  to  mediated  training 
in  the  specific  JPAS  skills  and  knowledges  required  to  effectively 
participate  in  the  system.  Elements  contributing  to  the  complexity  of  the 
problem  were  as  follows: 

1.  Size  and  heterogeneity  of  the  population. 

2.  Variability  of  media  resources  and  classroom  facilities  at  Air 
Force  Bases  where  the  training  is  to  be  conducted. 

3.  Unknown  capabilities  of  personnel  who  would  be  responsible  for 
delivering  training. 

4.  Resistance  to  the  CSRA.  Research  on  development  of  JPAS  indicated 
that  there  is  considerable  resistance  to  acceptance  of  the  new 
approach  as  mandated  by  the  law.  Therefore,  beyond  teaching  people 
to  use  the  system,  it  was  hoped  that  the  training  would  serve  as  a 
means  of  making  JPAS  more  acceptable  to  users. 
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1.  Supervisor  informs  employee  of  job  Performance 
A ppraisal  System . 

2.  Work  Plan  Meeting: 

—  Indenlify  Job  Performance  Elements  (JPEs) 

— ■  Dele  run  no  Eriliral  JPEs 

—  Assign  Relative  Importance  Points 

—  Set  Performance  Standards 

3.  Supervisor  signs  and  sends  Work  Plan  to  reviewer. 

4.  Reviewer  checks  Work  Plan. 

5.  1  ndicative  of  job/ 

0.  Meets  organizational  requirements? 

7.  Written  in  proper  format? 

8.  Reviewer  signs  and  returns  Work  Plan  to  supervisor 
for  employee's  signature. 

9.  Employee  and  supervisor  retain  a  copy  of  the  Work 
Plan. 

10.  Supervisor  completes  and  signs  appraisal  and  forwards 
it  to  reviewer. 

1  1.  Reviewer  checks  and  signs  appraisal. 

12.  Appraisal  returned  to  supervisor  to  obtain  employee's 
signature. 

13.  Returned  to  supervisor  for  reaccom plisbment. 

14.  Returned  to  supervisor  for  reaccompiishment. 


Figure  1,  Flowchart  of  JPAS  Activities 
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II. 


LITERATURE  REVIEW 


The  results  of  50  years  of  comparative  research  indicate  that  any 
medium  or  media  combination  can  teach  about  as  well  as  "traditional  instruc¬ 
tion."  This  is  a  meager  information  yield,  given  the  thousands  of  research 
studies  that  have  been  conducted.  Several  reviewers  (Allen,  1971;  Clark, 
1975;  Jamison,  Suppes,  &  Wells,  1974;  Solomon  &  Clark,  1977)  agree  that  the 
most  usual  kind  of  study,  in  which  the  scores  on  test  X  for  group  A  who 
received  treatment  A  are  compared  to  the  scores  of  group  B  who  received 
treatment  B,  simply  cannot  provide  information  on  those  aspects  of  a  treat¬ 
ment  which  are  particularly  important  ins truct ional ly .  In  general,  the  more 
traditionally  "well-controlled"  a  study  is,  the  more  likely  it  is  to  show  no 
difference  between  treatments  and,  equally  important,  thv  more  likely  it  is 
that  the  medium  employed  in  the  study  was  not  used  to  its  best  advantage. 

Two  consistent  findings  are  that  (a)  active  learner  response  during 
instruction  tends  to  improve  learning  (Campeau,  1974),  and  (b)  presenting 
the  learner  with  knowledge  of  the  correct  answer  after  a  response  also  tends 
to  increase  learning  (Reid,  1971).  Thus,  while  the  review  of  the  literature 
narrowed  the  field  of  possible  training  methods  to  those  which  are  active 
and  performance-based,  a  clear  picture  of  the  methods  which  would  be  most 
effective  did  not  emerge. 


III.  RESEARCH  PLAN 


Instructional  Materials 


Two  prototype  training  courses  were  developed,  differing  on  the  type  ot 
active  responding,  feedback  employed,  and  the  mode  by  which  information  was 
presented.  Each  was  designed  to  be  of  practical  use  to  the  student  and  to 
develop  an  attitude  of  acceptance  toward  the  system.  Each  covered  the  same 
information  and  had  the  same  objectives.  The  topics  included  in  the 
instruction  are: 


Unit  I. 
Unit  II. 
Unit  III. 
Unit  IV. 
Unit  V. 
Unit  VI. 
Unit  VII. 
Unit  VIII 
Unit  IX. 
Unit  X. 
Unit  XI. 


An  Overview  of  the  Air  Force  Civilian  Appraisal  System 
Job  Performance  Elements  (JPEs) 

Critical  JPEs 
Performance  Standards 
Priority  Weighting 
Work  Plan  Form 

Effective  Communication  and  the  Work  Plan  Meeting 
Role  of  the  Reviewer  in  the  Appraisal  System 
Periodic  Review 
JPAS  Ratings 
Post-Evaluation  Process 


Prototype  A  was  instructor-based.  That  is,  an  instructor  was  respon¬ 
sible  for  conducting  lectures,  demonstrations  and  group  discussions. 
Feedback  on  exercises  was  given  directly  by  the  instructor  or  through 
instructor-led  discussions.  It  was  designed  to  allow  opportunities  for 
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active  discussion  and  inquiry  about  the  system.  This  dynamic  process 
addressed  the  problems  associated  with  acceptance  of  the  system.  The 
materials  were  detailed  instructor  lesson  plans  with  overheads  and  a  student 
Workbook  containing  exercises  and  case  studies.  Instruction  took  place  in  a 
conventional  classroom  with  information  presented  by  the  instructor,  v*io 
also  led  the  class  through  a  series  of  application  exercises  contained  in 
the  Workbook. 

Prototype  B  was  self-instructional.  The  complexity  of  the  problem 
necessitated  a  solution  which  assured  easy  implementation  and  avoided  the 
requirement  for  locally  available  media  equipment.  Consequently,  a  pro¬ 
grammed  self-instructional  text  was  designed.  The  same  information  used  in 
the  instructor-led  course  was  contained  in  the  programmed  text.  The 
material  consisted  of  a  Resource  Book  and  a  Workbook.  Students  worked  at 
their  own  pace  reading  the  Resource  Book  and  applying  what  they  read  to 
exercises  in  the  Workbook. 


Dependent  Variables 

A  performance  test  was  developed  to  assess  how  well  each  prototype 
prepared  students  to  use  the  JPAS .  Measures  were  also  developed  to  evaluate 
students'  attitudes  toward  the  JPAS  as  a  function  of  the  training  prototype 
used  and  toward  the  instructional  prototypes  themselves. 

Performance  Test.  A  two-part  performance  test  was  developed  in  which 
students  had  to  apply  the  JPAS  for  three  different  job  holders  (a)  Job  1  ,  a 
service  station  attendant,  (b)  Job  2,  a  grocery  clerk,  and  (c)  Job  3,  a 
meter  reader.  These  jobs  are  obviously  not  Air  Force  positions,  but  they 
were  selected  for  their  general  familiarity.  Since  no  special  expertise  is 
required  to  understand  the  job  tasks,  it  was  assumed  that  the  content  of  the 
tests  was  not  biased. 

In  Part  1  of  the  performance  test,  the  task  was  to  write  job  perfor¬ 
mance  elements  (JPEs)  and  performance  standards  for  each  job.  The  materials 
used  for  this  task  were  (a)  a  position  description,  (b)  an  audiovisual 
presentation,  and  (c)  transcripts  of  interviews  with  both  the  employee  and 
supervisor.  In  Part  2  the  task  was  to  use  the  rating  system  to  evaluate 
each  job  holder's  performance  relative  to  the  standards.  The  materials  used 
here  were  a  hypothetical  work  plan  and  a  sample  rating  diary.  Since  both 
tasks  were  performed  for  each  of  the  three  jobs,  there  were  a  total  of  six 
subtests.  The  test  was  used  in  a  pilot  study  at  AFHRL  to  determine: 

(a)  if  the  instruments  could  be  easily  administered,  and 

(b)  if  the  materials  were  sufficient  in  content  to  permit  performance 
of  the  required  task. 

An  in-house  tryout  was  conducted  to  verify  usability  of  the  materials. 
Five  AFHRL  staff  members  participated  in  the  tryout  and  were  selected 
because  of  their  familiarity  with  the  JPAS.  Results  of  the  tryout  indicated 
that  the  package  of  materials  was  easy  to  administer  and  sufficient  in 
content  to  permit  development  of  a  feasible  work  plan.  Only  minor  revisions 
were  required. 


Checklists  were  developed  to  he  _0ed  in  scoring  the  work  plans.  Inter¬ 
rater  reliability  coefficients  vPearson  r)  for  three  persons  who  scored  the 
results  of  the  trvn.e  were  .98,  .93,  and  .92. 

Attitude  Scales.  An  attitude  instrument  consisting  of  a  semantic 
differential  scale  was  developed  to  ascertain  the  acceptability  cf  the  new 
system.  Students  were  asked  to  evaluate  the  training  programs  also. 
Likert-type  instruments  were  developed  to  assess  student  attitudes  toward 
the  prototypes.  They  were  administered  to  each  group  after  training.  On  a 
scale  from  1  to  7,  with  1  being  the  most  favorable,  students  were  asked  to 
place  an  "X"  on  the  line  at  the  point  which  reflected  how  they  felt  about 
the  course  on  16  dimensions.  The  scales  were  directed  at  assessing  the 
students'  general  interest  in  the  course  and  the  perceived  effectiveness  of 
specific  course  components. 

Copies  of  all  instruments  for  the  performance  test  and  attitude  scales 
are  presented  in  the  appendix.  With  minor  variations,  explained  later,  the 
same  instruments  were  used  in  each  stage. 


General  Procedures 


The  research  to  determine  the  more  effective  approach  was  conducted  in 
two  stages.  In  stage  one,  training  designs  using  the  two  prototypes  were 
tested  experimentally  at  Wright-Pat terson  AFB.  As  a  result  of  the  findings, 
changes  aimed  at  enhancing  the  teaching  effectiveness  were  made  in  the 
training  design.  Then  the  modified  prototypes  were  compared  in  the  second 
stage  at  Randolph  AFB.  Thus,  two  research  studies  were  carried  out  and  are 
described  separately  in  this  report. 


IV.  STAGE  ONE  COMPARISONS 


Independent  Variables 

Prototype  A  was  broken  out  into  two  approaches  according  to  the 
instructor's  knowledge  about  the  JPAS .  Two  types  of  instructors  were 
selected  to  administer  Prototype  A. 

1.  Instructors  who  were  unfamiliar  with  JPAS,  but  typically  would  be 
expected  to  conduct  the  training  at  Air  Force  bases  because  of 
their  availability.  They  were  labeled  "naive." 

2.  Instructors  who  were  subject  matter  experts  (i.e.,  highly  knowl¬ 
edgeable  about  JPAS  because  of  their  long-term  involvement  with  the 
system)  and  who  were,  therefore,  assumed  to  be  better  at  conveying 
accurate  information  about  the  system  than  the  naive  instructors. 
The  knowledgeable  instructors  were  labeled  Subject  Matter  Experts 
(SMEs) . 
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Consequently,  the  experimental  treatments  were  as  follows: 

Group  1.  Students  taught  by  naive  (non-SMI.)  instructors  using  the 
instructor-based  materials. 

Group  2.  Students  taught  by  SMEs  using  the  instructor-based  materials. 

Group  3.  Students  using  the  sel f- instruct ional  materials. 

The  research  hypothesis  was  that  the  performance  of  students  in  using 
the  course  materias  and  attitude  of  the  students  toward  both  the  JPAS  and 
the  training  course  itself  would  be  affected  differentially  by  the  type  of 
training  received. 


Procedures 

Instructor  Orientation.  In  order  to  learn  about  the  JPAS,  the  naive 
instructors  worked  through  the  self-instructional  course.  The  lesson  plans 
for  conducting  the  training  were  given  to  them  one  week  in  advance  of  the 
course.  They  were  told  to  review  the  materials  as  much  as  needed.  Finally, 
the  instructors  participated  in  a  question  and  answer  session  with  a  JPAS 
expert . 

No  training  was  offered  on  classroom  presentation  methods.  Both  the 
SMEs  and  the  naive  instructors  had  had  limited  teaching  experience.  In  this 
respect,  they  were  typical  of  the  kinds  of  personnel  who  might  be  expected 
to  teach  a  JPAS  course. 

Administration  of  Training  Course.  The  Stage  One  field  test  was 
conducted  at  Wright-Patterson  AFB.  Instructor-based  sessions  were  completed 
in  two  days  (16  hours)  of  training.  Self-instructional  students  worked  at 
their  own  rate.  The  dependent  measure  was  administered  to  all  groups 
simultaneously  on  day  three. 

Subjects 

The  field  test  population  consisted  of  141  persons  randomly  sampled 
from  the  upper  GS  levels  (a  majority  were  GS  10  or  above),  some  of  whom  were 
supervisors.  The  sample  contained  no  FWS  or  union  personnel.  The  subjects 
included  engineers,  accountants,  contract  specialists,  and  others  in  the 
professional,  technical,  and  managerial  categories. 

The  field  test  population  was  assigned  to  three  experimental  treat¬ 
ments.  Group  1  consisted  of  two  classes  led  by  "naive"  instructors  (n=42); 
Group  2  consisted  of  two  classes  led  by  SMEs  (n=39);  and  Group  3  was  the 
sel f-instruct ional  (SI)  group  (n=60).  Population  figures  represent  the 
maximum  number  of  students  for  which  data  are  available. 
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Data  Analysis 


Data  were  analyzed  using  the  Analysis  of  Variance  (ANOVA)  for  each  of 
the  six  performance  subtests  and  attitude  scale.  When  significant  differ¬ 
ences  were  found,  the  t-test  for  differences  among  several  means  was  used  to 
identify  the  source  of  the  difference.  The  confidence  level  was  set  at  .05. 

Due  to  student  attrition,  which  was  unrelated  to  the  treatment,  the 
total  Ns  for  the  dependent  variable  subtests  varied.  Reasons  for  different 
Ns  in  Stage  One  were: 

Participation  was  voluntary.  Students  could  choose  not  to  attend 
the  entire  testing  session. 

Student  fatigue.  Students  failed  to  complete  all  parts  of  the 
test.  This  was  especially  true  of  Part  2  of  the  dependent 
variable.  Notice  that  Job  3  of  Part  2  has  the  smallest  N  (123). 
This  was  the  last  task  to  be  completed  in  all  cases. 

-  Weather  conditions.  Icy  roads  caused  several  students  to  be  late 
for  the  test.  Thus,  data  were  unavailable  for  those  jobs 
administered  early  on  the  test  day. 

Performance  Test  Results.  Significant  differences  were  found  on  five 
of  the  six  subtests.  This  supports  the  hypothesis  of  differential  effects 
on  performance  as  a  function  of  training  design.  Tables  1  and  2  present  the 
ANOVA  results  for  each  compariron  and  Table  3,  the  means  and  standard 
deviations . 


Table  1.  Stage  One  ANOVA  Results  for  Performance  Subtests 
Job  1:  Part  1  Performance  Standards 


Source 

SS 

df 

ms 

F 

P 

Between  Groups 

mmaxm 

mm 

7.32 

<.001 

Within  Groups 

HEH 

— 

— 

Total 

liKMI 

m 

HE SI 

— 

— 

Job  2 :  Part  1  Performance  Standards 


Source 

SS 

df 

ms 

F 

P 

Between  Groups 

65 

3.34 

<.001 

Within  Groups 

1,298 

■9 

■ 

— 

— 

Total 

1,363 

135 

Hi 

— 

— 

Job  3:  Part  1  Perfor 

mance  Standards 

Source 

SS 

df 

ms 

F 

P 

Between  Groups 

5.20 

<.01 

Within  Groups 

WBm 

— 

— 

Total 

■Kn 

HU 

— 

— 

8 


r 

f 


Table  2.  Stage  Two  ANOVA 

Results  for  Performance  Subtests 

Job  1:  Part  2  Rating  Performance 

Source 

SS 

df 

ms 

F 

P 

Between  Groups 

182 

2 

90.75 

5.78 

<.005 

Within  Groups 

2,120 

135 

15.70 

Total 

2,301 

137 

_ ~ _ 

Job  2:  Part  2  Rating  Performance 

Source 

SS 

df 

ms 

F 

P 

Between  Groups 

16 

2 

8.12 

.47 

NS 

Within  Groups 

2,297 

136 

17.14 

— 

Total 

2,313 

138 

— 

-- _ 1 

Job  3:  Part  2  Rating  Performance 

Source 

SS 

df 

ms 

F 

P 

Between  Groups 

84 

2 

42.10 

5.38 

<•01 

Within  Groups 

939 

120 

7.83 

- 

Total 

1,023 

*2 2 

- 

- 

The  t-test  comparisons  to  determine  which  means  differed  significantly 
revealed  that  naive  instructors  had  a  detrimental  effect  on  performance. 
See  Tables  4  and  5  for  t-test  results.  Students  trained  by  SMEs  performed 
significantly  better  than  naive  instructors'  students  on  all  conditions 
where  a  significant  difference  was  found.  Students  using  the  self- 
instructional  materials  performed  significantly  better  than  students  taught 
by  naive  instructors  on  four  of  the  subtests. 

In  comparing  the  performance  of  the  self-instructional  students  with 
students  led  by  SMEs,  no  significant  difference  on  the  six  subtests  was 
found.  The  data  indicate  that  students  working  alone  do  as  well  as  students 
led  by  SMEs. 


Table  3.  Stage  One  Group  Means  and  Standard  Deviations 
on  the  Performance  Test 


Part  1.  Rating  Performance  Standards 


Job  1 

Job  2 

Job  3 

X 

SD 

X  SD 

X  SD 

■Group  1 
(Naive) 

15.02 

3.67 

16.24  2.83 

14.90 

3.41 

Group  2 

( SME- Ins true tor 
based) 

17  .42 

2.69 

17.92  3.60 

17.36 

3.24 

Group  3 
(Self-Instr . ) 

17.05 

2.97 

17.60  3.75 

16.03 

3.41 

Scoring  of  Performance  Standards: 

5  points  given  for  JPEs. 

1  point  given  for  identification  of  a  critical  JPE. 

13  points  given  for  performance  standards. 

3  points  given  for  assigning  relative  importance  points. 
Total  possible  score:  22  points 

Part 

2.  Rati: 

ng  Performance 

Job  1 

Job  2 

Job 

3 

X 

SD 

X  SD 

X 

SD 

Group  1 
(Naive) 

12.68 

4.37 

13.15  4.28 

10.79 

2.87 

Group  2 
(Knowl . ) 

15.50 

3.66 

13.89  3.76 

12.17 

2.54 

Group  3 
(Self-In. ) 

14.90 

3.85 

13.92  4.28 

12.80 

2.91 

Scoring  of  Rating  Task 

7  points  given  for  correctly  using  rating  form. 

2  points  given  for  each  correct  standard's  rating. 
Total  score  Job  1-25  points 
Total  score  Job  2-27  points 
Total  score  Job  3-21  points 
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t-Test  Results 


The  t-test  was  used  to  determine  which  specific  means  differed 
significantly  from  each  other  after  an  F-ratio  was  found.  In  all  instances, 
if  the  difference  between  any  two  means  is  larger  than  the  critical 
difference,  then  the  means  are  assumed  to  be  significantly  different.  The 
critical  difference  is  the  product  of  the  standard  error  of  the  difference 
among  means  and  the  t-value  for  the  appropriate  degrees  of  freedom  and  the 
alpha  level  of  significance. 


Table  4.  Stage  One:  Part  I  Performance  Standards  Test  Results 

t-test  for  Job  1 


Group  2  Group  3  Critical  diff. 

X  =  17.42  X  =  17.05  (at  .05  level) 


Group 

1  : 

X 

5= 

15.02 

2.40* 

2.03* 

1.30 

Group 

2: 

X 

17.42 

— 

.37 

1.30 

t-test 

for  Job  2 

Group  2 

Group  3 

Critical  diff. 

X  =  17.92 

X  =  17.60 

(at  .05  level) 

Group 

1  : 

m 

1.68* 

1.36* 

1.33 

Group 

2: 

m 

BkwI 

— 

.32 

1.33 

t-test 

for  Job  3 

Group  2 

Group  3 

Cr i t ical  diff. 

X  =  17.36 

X  «  16.03 

(at  .05  level) 

Group 

1: 

X 

= 

ilKBXKiil 

2.46* 

1.41 

Group 

2: 

X 

— 

1  .41 

Table  5.  t-Test 

for  Stage  One: 

Part  2.  Rating  Performance  Results 

t-test 

-  Job  1 

Group  2 

Group  3 

Cri t ical  diff. 

X  =  15.50 

X  =  14.90 

(at  .05  level) 

Group 

1: 

X  - 

12.68 

2.82 

2.22 

1 .66 

Group 

2: 

X  = 

15.50 

— 

.60 

1 .66 

t-test 

-  Job  3 

Group  2 

Group  3 

Critical  diff. 

X  *  12.17 

X  -  12.80 

(at  .05  level) 

Group 

1 : 

1.38 

2.01 

1.26 

Group 

2: 

toil 

— 

.63 

1.26 
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Attitude  Toward  System  Results.  The  type  of  training  received  had  no 
effect  on  student  s '  attitude  toward  the  system.  Means  and  standard 
deviations  are  presented  in  Table  6.  On  a  scale  from  1  to  7,  with  7  being 
most  favorable,  the  means  were  4.46,  3.86,  and  4.01,  for  the  three  groups 
respectively.  Table  7  presents  the  ANOVA  results.  Because  there  is  no 
significant  difference,  the  hypothesis  that  training  designs  would  have  a 
differential  effect  on  attitude  toward  the  JPAS  was  rejected. 


Table  6.  Groups  Means  and  Standard  Deviations  for 
Attitude  Toward  JPAS 


N 

X 

SD 

Group  1 

42 

4.46 

1  .67 

Group  2 

39 

3.86 

1.29 

Group  3 

54 

4.01 

1.24 

Table  7.  Stage  One  ANOVA  Results  for  Attitude  Toward  JPAS 


Source 

SS 

L  df 

ms 

F 

P 

Between  Groups 

8.71 

|g|E|^ 

fjggMM 

2.87 

NS 

Within  Groups 

201.29 

— 

Total 

210 

■H 

■B 

— 

Attitude  Toward  Training  Results.  Table  8  presents  the  means  and 
standard  deviations  for  each  group.  A  significant  difference  was  found  in 
the  student  evaluation  of  training.  Table  9  presents  the  ANOVA  results. 


Table  8.  Stage  One:  Group  Means  and  Standard  Deviations 
for  Attitude  Toward  Training 


Group 

N 

X 

SD 

1  -  Naive  Instr. 

37 

3.55 

.97 

2  -  SME  Instr. 

38 

3.22 

.82 

3  -  Self-Instr. 

60 

3.89 

.97 

Table  9.  Stage  One:  ANOVA  Results  for  Attitude  Toward  Training 


Source 

SS 

df 

ms 

F 

P 

Between  Groups 

10.69 

5.34 

6.07 

<.005 

Within  Groups 

115.57 

■51 

.88 

— 

— 

Total 

126.26 

■Hi 

— 

— 

2 


The  t-test  comparisons  showed  that  students  in  Group  2  (SME)  had  a 
more  favorable  perception  of  the  training  than  did  the  students  in  Group  3 
(self-instructional).  See  Table  10.  Significant  differences  for  naive 
instructor-led  groups  and  the  self-instructional  group  were  not  found. 


Table  10.  Stage  One:  t-Test  for  Attitude  Toward  Training 


Critical 

Group  2 

Group  3 

Difference 

X  =  3.22 

X  =  3.89 

(at  .05  Level) 

Group  1: 

X  =  3.55 

.33 

.34 

.40 

Group  2: 

X  =  3.22 

— 

.67* 

.40 

The  hypothesis  that  training  prototype  affects  student  attitude  toward 
the  course  itself  was  confirmed.  Students  seem  to  prefer  a  course  conducted 
by  an  SME  over  a  sel f-instructional  package  (p<.005).  This  preference  is 
not  obtained  when  the  instructor  is  not  an  SME. 


V.  STAGE  TWO  COMPARISONS 


Independent  Variables 


The  prototypes  evaluated  in  this  stage  were  enhanced  versions  of  those 
tested  in  Stage  One.  Specific  enhancement  strategies  were  as  follows: 

1.  The  instructor-based  and  self-instructional  prototypes  were  revised 
to  overcome  learning  difficulties  common  to  both.  In  addition,  the 
JPAS  rating  procedures  had  been  changed  by  AFHRL  in  the  interim, 
and  the  course  was  revised  to  accommodate  the  new  procedures. 

2.  The  naive  instructor  training  design  was  eliminated  on  the  basis  of 
Stage  One  results. 

3.  The  SME-led  training  design  was  modified  to  add  preparation  on 
teaching  skills  and  a  more  in-depth  explanation  of  the  JPAS  to  the 
instructors.  All  instructors  in  Stage  One  had  expressed  some 
anxiety  about  their  delivery  skills.  Therefore,  a  one-hal f  day 
training  session  was  provided  to  allow  practice  and  feedback  on 
presentation  methods  and  group  facilitating.  The  hypothesis  was 
that  assistance  on  teaching  skills  would  result  in  better  student 
performance  and  attitude. 

4.  A  one-half  day  small  group  exercise  on  applying  JPAS  to  a  job  was 
developed  to  be  used  after  students  had  completed  the  self- 
instructional  package.  The  hypothesis  was  that  the  addition  of  an 
opportunity  for  group  discussion  would  result  in  better  student 
performance  and  attitude. 
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Consequently,  the  experimental  treatments  were: 

Group  1.  Students  using  the  basic  self-instructional  package. 


Group  2.  Students  using  the  self-instructional  package  with  a  one-half 
day  group  application  exercise. 

Group  3.  Students  taught  by  SMEs  who  had  participated  in  an  instructor 
training  session  and  used  the  instructor-based  package. 

The  general  research  hypothesis  was  that  performance  and  attitude  of 
the  students  would  be  differentially  affected  by  the  type  of  training 
received . 


Dependent  Variables 

Dependent  measures  were  the  same  two-part  performance  test  and  the 
attitude  scales  used  in  Stage  One.  However,  to  reduce  testing  time,  the 
performance  test  consisted  of  applying  the  JPAS  to  two  jobs  only:  the 
service  station  attendant  and  the  meter  reader.  Since  the  JPAS  rating 
procedures  had  changed,  a  new  scoring  mechanism  was  devised  for  Part  2  of 
the  performance  test. 


Procedures 


The  one-half  day  training  session  for  instructors  was  conducted  the  day 
before  the  sessions  began.  The  field  test  was  conducted  at  Randolph  AFB . 

Two  days  were  allotted  for  instruction,  followed  by  test  administration  on 
the  third  day. 


Subjects 

The  subjects  wire  92  workers  and  supervisors  randomly  drawn  from  the 
lower  GS  levels  and  tssigned  to  the  experimental  treatments.  Subjects  were 
mainly  clerical  and  ’WS  employees.  Group  1  (n=29)  received  the  self- 
instructional  training.  Group  2  (n=18)  received  the  self-instructional 
training  and  group  application  exercise,  and  Group  3  (n=45)  received  the  SME 
training.  Population  figures  represent  the  maximum  number  of  students  for 
which  data  are  available. 


Data  Analysis 

Data  were  analyzed  using  the  ANOVA  for  each  of  the  four  subtests  and 
the  attitude  scales.  When  significant  differences  were  found,  the  t-test 
for  differences  among  several  means  was  used  to  identify  the  source  of  the 
differences.  The  confidence  level  was  set  at  .05. 

Due  to  student  attrition,  which  was  non-treatment  related,  the  total 
N's  for  the  dependent  variable  components  vary. 


/ 


Performance  Test  Results.  Performance  test  results  showed  no  signifi¬ 
cant  differences  in  the  ability  of  students  to  use  the  JPAS.  The 
enhancements  of  each  prototype  had  no  effect  on  learning  compared  to  the 
basic  se l f- i ns t r uct ional  prototype.  Therefore,  the  research  hypothesis  of 
differential  effects  was  rejected.  As  in  Stage  One,  the  self-instructional 
course  and  the  SME-led  course  were  equally  effective.  Table  11  presents 
the  ANOVA  results,  and  Table  12  the  means  and  standard  deviations. 


Table  11.  Stage  Two  ANOVA  Results  for  Performance 


Job  1:  Part  1  Performance  Standards 


|  Source 

SS 

df 

ms 

F 

P 

|  Between  Groups 

00 

r-'. 

O' 

2 

4.89 

.52 

NS  | 

Within  Groups 

785.81 

83 

9.47 

- 

1 

|  Total 

795.59 

L_,  AL-.] 

_ 11 _ 

- 

_ 11 _ 1 

Job  2:  Part  1  Performance  Standards 


Source _ 

Between  Groups 
Within  Groups 
Total 


SS  I 
12.21  | 
980.16  | 

992.37  | 


d_f 

2 

83 

85 


ms  | 
6.10  | 
11.81  | 


F 


.52 


J _ P 

NS 


Job  1:  Part  2  Performance  Standards 


Source 

SS 

df 

ms 

t 

P _ 1 

Between  Groups 

18.24 

2 

9.12 

.15 

1  NS  I 

Within  Groups 

5 ,036.76 

81 

62.18 

1  -  1 

Total 

5,055.00 

83 

— 

J _ 11 _ 1 

Job  2:  Part  2  Performance  Standards 


Source 

SS 

df 

ms 

F 

P_  _ 

Between  Groups 

|  77.22  S 

2 

|  38.61 

.38 

NS  j 

Within  Groups 

|  7,859.58  j 

77 

|  102.07 

I 

Total 

|  7,936.8  j 

79 

J _ 11 _ 

1 

15 


Table  12.  Stage  Two  Group  Means  and  Standard  Deviations 
on  the  Performance  Test 

Part  1.  Rating  Performance  Standards 


Job  1  Job  2 


X 

SD 

X 

SD 

Group  1 

( Sel f-Instruct ional ) 

15.41 

3.13 

15.86  3.82 

Group  2 

(Self-instructional 
plus  group 
appl icat ion 

16.29 

3.60 

17.00  4.07 

Group  3 

(SME- Ins true tor) 

16.05 

2.85 

16.27  2.94 

Scoring  of  Performance  Standards: 

5  points  given  for  JPEs . 

1  point  given  for  identification  of  a  critical  JPE. 

13  points  given  for  performance  standards. 

3  points  given  for  assigning  relative  importance  points 
Total  possible  score:  22  points 

• 

Part  2. 

Rating  Performance 

Job  1 

Job  2 

X 

SD 

X  SD 

Group  1 

(Self-Instructional) 

24.85 

8.36 

13.38  10.45 

Group  2 

(Self-Instructional 
plus  group 
application) 

25.53 

8.13 

25.43  11.26 

Group  3 

(Self-Instructor) 

25.90 

7.48 

25.50  9.45 

Scoring  of  Rating  Task 

7  points  given  for  correctly  using  rating  form. 

2  points  given  for  each  correct  standard's  rating. 
Total  score  Job  1-25  points 

Total  score  Job  2-27  points 

Total  score  Job  3-21  points 


1 
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Attitude  Results.  As  in  Stage  One,  results  showed  that  type  of 
training  had  no  effect  on  attitude  toward  JPAS .  Table  13  presents  the  ANOVA 
results.  Table  14  presents  the  means  and  standard  deviations. 


Table  13.  Stage  Two  ANOVA  Results  for  Attitude  Toward  the  JPAS 


Source 

SS 

df 

ms 

F 

P 

Between  Groups 

3.26 

2 

1.26 

NS 

Within  Groups 

108.01 

84 

— 

— 

Total 

111.27 

86 

■sR 

— 

— 

Table  14.  Group  Means  and  Standard  Deviations  for 
Attitude  Toward  JPAS 


N 

X  ! 

SD 

Group  1 

29 

1.25 

Group  2 

15 

1 

.99 

Group  3 

43 

■KH 

1.10 

The  evaluation  of  training  results  revealed  that  students  preferred  tne 
SME-led  course  over  both  of  the  other  types  of  training  courses  adminis¬ 
tered.  Table  15  presents  the  means  and  standard  deviations  ( l=favorab le , 
7=unfavorable) ,  Table  16  presents  the  ANOVA  results.  Table  17  shows  t-test 
results. 


Table  15.  Stage  Two:  Group  Means  and  Standard  Deviations 
for  Attitude  Toward  Training 


Group 

N 

X 

SD 

1 

-  Self  Instr. 

29 

3.06 

1.11 

2 

-  Sel f-Instr . / 
Small  Group 

18 

3.33 

.79 

3 

-  SME  Instr. 

45 

2.55 

.85 

Table  16.  Stage  Two  ANOVA  Results  for  Attitude  Toward  Training 


Source 

SS 

df 

ms 

F 

P 

Between  Groups 

2 

8.37 

<.001 

Within  Groups 

• 

89 

.98 

— 

— 

Total 

103.97 

91 

— 

— 

— 

1 


l 
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Table  17.  t-Test  for  Stage  Two:  Attitude  Toward  Training 


Here  the  Ns  were  disparate.  That  is,  there  were  more  than  twice  as  many 
in  the  largest  group  as  in  the  smallest  group  (Group  1  N  was  18  while  Group  3 
N  was  45).  Therefore,  the  standard  error  was  computed  separately  for  each 
comparison.  The  results  are  presented  below. 


Group  2 

vs.  Group 

1 

(c 

diff. 

=  .60) 

3.33 

3.06 

— 

.17 

(non-signif icant) 

Group  2 

vs.  Group 

3 

(c 

diff. 

=  .54) 

3.33 

2.55 

— 

.78 

(significant) 

Group  1 

vs.  Group 

3 

(c 

diff. 

=  .48) 

3.06 

2.55 

* 

.51 

(significant) 

VI.  GENERAL  DISCUSSION  AND  CONCLUSIONS 
Naive  Instructors  vs.  SMEs 


In  and  instructor-based  system,  subject  matter  expertise  appears  to  be 
important  in  training  people  to  use  the  JPAS.  Instructors  who  were  labled 
naive  had  a  detrimental  effect  on  learning  despite  their  precourse  orientation 
to  the  JPAS  and  the  use  of  highly  detailed  lesson  plans.  Observations  and 
post-hoc  analysis  of  the  naive  instructors'  behavior  indicate  that  they  were 
less  able  to  field  questions  about  the  system  and  that  they  introduced  their 
misconceptions  about  the  system.  Students'  attitudes  toward  the  training  is 
congruent  with  the  finding  that  the  subject  matter  expertise  is  a  significant 
factor,  as  students  taught  by  SMEs  were  more  favorably  inclined  toward  the 
training  than  those  taught  by  non-SMEs. 

Clearly,  on  the  basis  of  performance  and  attitude  toward  the  training, 
field  test  results  showed  that  if  an  instructor-based  program  is  used  to  teach 
the  JPAS,  the  instructor  should  be  an  SME. 

The  SMEs  employed  in  this  study  were  drawn  from  the  AFHRL  at  Brooks  AFB 
and  the  contractor  (Applied  Science  Associates)  employees  who  participated  in 
development  of  the  course.  All  of  these  people  had  been  working  with  the  JPAS 
for  several  months  (i.e.,  from  4  months  to  more  than  1  year).  Thus,  their 
indoctrination  was  long  and  intensive. 

If  an  instructor-based  system  is  selected,  a  training  program  which 
thoroughly  indoctrinates  instructors  in  the  use  of  the  JPAS  must  be 
developed.  The  self-instructional  package  used  to  train  the  naive 
instructors  in  the  Wright-Patterson  field  test  was  obviously  inadequate.  Just 
how  much  more  training  and  what  type  of  instruction  is  necessary  to  produce 
results  similar  to  those  achieved  in  the  field  test  remains  an  empirical 
question.  One  approach  to  determining  the  nature  of  a  JPAS  instructor 
training  package  is  to  perform  an  analysis  of  the  tasks  performed  by  personnel 
in  the  process  of  becoming  SMEs.  In  this  way,  the  experiences  necessary  to 
becoming  an  SME  can  be  identified  and  training  can  be  designed  to  simulate 
these  conditions. 
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Se 1 f- Ins t rue t iona 1  vs.  SME-led  Training 


There  are  no  differences  in  student  performance  or  attitude  toward  the 
system  between  students  using  the  self-instructional  package  and  those 
taught  by  SMEs^.  However,  students'  attitudes  toward  the  training  is  more 
favorable  when  instruction  is  presented  by  an  SME.  While  the  difference  is 
statistically  significant,  the  difference  of  about  one  half  a  point  (.51) 
must  be  evaluated  in  terms  of  practical  implications. 

On  a  scale  of  one  to  seven,  with  one  being  the  most  favorable,  the  mean 
of  the  se 1 f-ins t r uct ional  group  was  3.06,  which  is  on  the  moderate  to 
favorable  side  of  the  scale  as  is  the  SME-led  group  with  a  mean  of  2.55. 
The  evidence  reveals  no  real  dislike  for  the  self-instructional  approach, 
but  clearly  the  SME-led  training  is  preferred.  But,  since  this  slight 
preference  for  SME-led  training  does  not  translate  into  better  performance 
and  acceptance  of  the  system,  the  importance  of  this  difference  is 
questionable. 

In  comparing  the  practical  benefits  of  the  two  prototypes,  however, 
there  are  certain  administrative  considerations  which  do  appear  to  make  a 
difference  in  terms  of  time  and  resource  allocations. 

Figure  2  shows  the  completion  times  for  the  students  who  participated 
in  the  self-instructional  sessions  in  Stage  One.  At  the  end  of  the  first 
day,  33  percent  of  the  students  turned  in  the  instructional  materials  and 
returned  to  their  job.  By  10  a.m.  of  the  second  day  (after  10  instructional 
hours)  an  additional  40  percent  were  ready  to  return  to  work.  All  of  the 

students  turned  in  the  materials  within  14  hours.  The  median  completion 

time  was  nine  hours.  Obviously,  all  students  in  the  group-paced, 

instructor-led  training  finished  at  the  same  time  (after  16  hours).  As 

shown  in  Figure  3,  essentially  the  same  effect  was  found  in  the  Stage  Two 
test  at  Randolph  AFB.  That  is,  50  percent  of  the  students  were  finished 
after  nine  hours,  and  everyone  was  finished  after  11.5  hours. 

If  time  away  from  the  job  is  a  factor,  clearly  the  self-instructional 
approach  is  preferable  since  students  achieved  the  same  performance  level  as 
with  the  instructor-based  training,  but  were  returned  to  work  sooner.  This 
is  a  significant  factor  when  large  numbers  of  personnel  are  being  trained. 

The  self-instructional  package  can  be  administered  by  almost  anyone, 
whereas  the  instructor-led  course  requires  an  SME  as  the  instructor.  As 
discussed  above,  all  potential  JPAS  instructors  should  become  experts  in  the 
system  and  become  familiar  with  the  detailed  lesson  plans  for  conducting  the 
training.  Once  the  course  is  underway,  the  level  of  effort  for  actual 
administration  of  the  self-instructional  package  is  lower  for  two  reasons: 
(a)  the  self-instructional  administrators  finish  as  soon  as  the  last  student 
has  turned  in  his  materials,  approximately  12  to  14  hours,  while  SME 

^Th  i  s  study  did  not  take  the  reading  ability  into  account.  Subjects  were 
drawn  from  the  populations  at  the  bases  and  randomly  assigned  to  treatments. 
No  measures  of  reading  ability  were  given.  Therefore,  the  effectiveness  of 
these  packages  for  individuals  with  low  reading  ability  is  not  known. 
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Percent  of  Students  Remaining  in  Class 


figure  2-  Self -Instructional  Completion  Time 
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Self- Instructional  Completion  Time 


PERCENT  OF  STUDENTS  REMAINING  IN  CLASS 


Figure  3.  Stage  Two. 


Completion  Times  of  Self-Instructional  Students 


r 


instructors  require  16  hours  to  administer  the  course,  and  (b)  the  self- 
instructional  administrators  can  monitor  35  or  more  students  with  ease, 
while  the  SME  instructors  should,  according  to  the  course  design,  teach  no 
more  than  25  persons. 

Certainly,  the  demand  on  human  resources  for  the  instructor-led  course 
is  significant  when  thousands  of  persons  must  be  trained.  Therefore, 
administratively  and  in  terms  of  cost-effectiveness,  the  self-instructional 
package  is  preferable. 
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JRSCEDIN3  Pa(JS  BLaNK-iN  OT  FliWH) 


Think  about  the  Joh  Performance  Appraisal  System  process.  Then  for 
each  pair  of  words  listed  below,  put  an  "X"  on  the  line  at  the  point 
which  reflects  how  you  feel  about  it.  You  should  record  an  "X"  for 
each  pair  of  words. 

.JOB  PERFORMANCE  EVALUATION  SYSTEM 


GROUP  ll 


Evaluation  of  the  Training 
Self-Instructional 


This  evaluation  is  intended  to  find  out  about  your  attitude  toward 
the  course.  Read  each  item  and  place  an  "X"  in  the  line  at  the  point 
which  reflects  how  you  feel  about  it.  You  should  record  an  "X"  for  each 
pair  of  words. 


1. 

How  did  the  course  compare 

to  other  training 

you  have 

taken? 

Good  1 

:  2  : 3 

:4 

:  3 

:  6 

:  7  : 

Bad 

2. 

How  interesting 

was  the  course  in 

general? 

Interesting  1 

:  2  : 3 

:  4 

:5 

:  6 

:  7  : 

Boring 

3. 

How  often  would 

you  like  to 

have 

courses 

like 

this  one 

again' 

? 

Frequently  1 

:  2  : 3 

:  4 

:  3 

:  6 

:  7  : 

Rarely 

4. 

How  well  did  the  course  prepare  you  to  a 
JPAS  system  on  the  job? 

ppiy 

the  elements  of 

the 

a.  Job  Performance  Elements? 

Very  Well  1 

:  2  : 3 

:  4 

:  3 

:  6 

:  7  : 

Very 

poorly 

b.  Critical  Elements? 

Very  Well  1 

:  2  : 3 

:  4 

:  3 

:  6 

:  7  : 

Very 

poorly 

c.  Performance 

Standards? 

Very  Well  I 

:  2  : 3 

:  4 

:  5 

:  6 

:  7 

Very 

poorly 

d.  Priority  Weights? 

Very  Well  1 

:  2  : 3 

:  4 

:  5 

:  6 

:  7  : 

Very 

poorly 
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Self-Instructional,  continued 


5.  How  well  did  the  course  prepare  you  to  complete  the  forms  used 
In  the  system? 

a.  Work  Plan  Form? 

Very  Well  1 _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ :  Very  poorly 

b.  Quarterly  Work  Plan  Progress  Review? 

Very  Well  1 _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ :  Very  poorly 

c.  JPAS  Rating  Form  (yearly)? 

Very  Well  1 _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ :  Very  poorly 

6.  How  effective  were  the  following  materials  in  helping  you  learn? 

a.  Practical  exercises  in  the  workbook? 

Effective  1 _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ :  Ineffective 

b.  Case  Studies  in  Appendix  A  of  the  Workbook? 

Effective  1 _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ :  Ineffective 

7.  How  often  did  you  have  questions  that  the  manual  did  not  address? 

Very  often  1 _ :2 _ :3 _ ;4 _ :5 _ :6 _ :7 _ :  Rarely 

8.  How  sufficient  were  the  answers  to  the  exercises? 

Sufficient  1 _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ :  Insufficient 

9.  How  well  did  you  like  working  independently? 

Very  much  1 _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ :  Not  at  all 

10.  How  clear  were  the  directions  for  the  activities  in  the  manual? 

Very  Clear  1 _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ :  Very  Unclear 
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GROUP  It 


Evaluation  of  the  Training 
Group-Paced 

This  evaluation  is  intended  to  find  out  about  your  attitude  toward 
the  course .  Read  each  item  and  place  an  "X"  in  the  line  at  the  point 
which  reflects  how  you  feel  about  it.  You  should  record  an  "X"  for  each 
pair  of  words. 

1.  How  did  the  course  compare  to  other  training  you  have  taken? 

Good  1 _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ :  Bad 

2.  How  interesting  was  the  course  in  general? 

Interesting  1 _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ Boring 

3.  How  often  would  you  like  to  have  courses  like  this  one  again? 

Frequently  I _ :2 _ :3 _ :4 _ :5 _ :6 _ :7 _ :  Rarely 

4.  How  well  did  the  course  prepare  you  to  apply  the  elements  of  the 
JPAS  system  on  the  job? 

a.  Job  Performance  Elements? 


Very  Well  1 

:  2 

:  3 

:4 

:  5 

:b 

:  7 

:  Very 

poorly 

b.  Critical 

Very  Well  1 

Elements? 

:  2 

:  3 

:4 

:  5 

:  6 

:  7 

:  Very 

poorly 

c.  Performance  Standards? 

Very  Well  1  :2  : 3 

:  4 

:5 

:  6 

:  7 

:  Very 

poorly 

d.  Priority 

Very  Well  1 

We  ights? 

:2 

:  3 

:4 

:  5 

:  6 

:  7 

:  Very 

poorly 
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Croup-Paced,  continued 


5.  How  well  did  the  course  prepare  you  to  complete  the  forms  used  in 
the  system? 

a.  Work  Plan  Form? 


Very  Well  1 : 

:  2  : 3  :4 

:5 

:  6 

:  7  ; 

Very  Poorly 

b.  Quarterly  Work  Plan  Progress 

Rev i ew? 

Very  Well  1 

:2  : i  :4 

:5 

:  6 

:  7  ; 

Very  Poorly 

c,  JPAS  Rating 

Form  (yearly)? 

Very  Well  1 

:  2  :  '1  ;  4 

:  5 

:  6 

:  7  : 

Very  Poorly 

How  effective  were  the  following 

materials  in 

helping 

you  learn? 

a.  Practical  exercises  in  the  workbook? 

Effective  1 

:  2  :  1  : 4 

:  5 

:b 

:  7  ; 

Ine  f  fee  t  i  ve 

b.  Case  Studio; 

■i  in  Appendix  A  of 

the  Workbook 

? 

Effective  1 

;  2  :  )  :4 

:  3 

:  6 

:  7  : 

Inel fective 

7.  How  effective  were  the  following  in  support ing  the  instruction? 


a.  Instructor's  presentation? 

Very 

Ineffective  1  :2  :  )  :4 

:  5 

:  b 

:  7 

Very 

:  Effective 

b.  Group  interaction? 
Very 

Ineffective  1  :2 

:3  :4 

:  5 

:  b  : 

:  7 

Very 

:  Effective 

c.  Your  participation? 
Very 

Inef  feet ive  1  :  2 

:  5  ;4 

:  5 

:  6  : 

;  7 

Very 

:  Effective 

d.  Audio-Visual  aids? 
Very- 

Ineffective  1  :2 

:  6  ; 

;  7 

Very 

:  Effective 
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